Distributed Real-Time Chunking

Adds support for Distributed Real-Time Chunking (DRTC). DRTC extends RTC to a distributed client-server setup. You can think of it as combining RTC's in-painting with async inference's networked client-server pattern.

Demonstration videos of DRTC in action can be found here

Type / Scope

• Type: Feature. Adds support for DRTC
• Scope: This is mostly additive with some minor non breaking changes to the async inference gRPC contract.

Summary / Motivation

• Adds support for Distributed Real-Time Chunking (DRTC). DRTC extends RTC to a distributed client-server setup. You can think of it as combining RTC's in-painting with async inference's networked client-server pattern.
• The current RTC implementation does not support a distributed client server setup. The current async inference implementation does not support RTC in-painting which is needed for smooth inference when running async action chunking. DRTC closes this gap. Furthermore, DRTC is robust to network failures modes include lost observations and actions, duplicated messages, and re-ordered messages.

What changed

Main Files Added

• src/lerobot/async_inference/robot_client_drtc.py - DRTC equivalent of Async Inference robot_client.py
• src/lerobot/async_inference/policy_server_drtc.py - DRTC equivalent of Async Inference policy_server.py
• src/lerobot/async_inference/configs_drtc.py - DTTC equivalent of Async Inference config.py
  - src/lerobot/async_inference/utils - DRTC specific utilities
• src/lerobot/async_inference/rtc_guidance.py RTC guidance for DRTC, mostly copied from src/lerobot/policies/rtc/modeling_rtc.py, also includes Alex Soar's RTC improvements from RTC adjustments. Bug fix & Alex Soare optimization #2499
• Other file changes are mostly my experiment tracking, trajectory visualization, and tests

How was this tested (or how to run locally)

• Ran the client + server on the same node setup with:

# Start the policy server and client and run the experiment as described in the baseline.yaml config
./scripts/run_drtc_experiment.sh --config baseline.yaml

• Ran distributed client server setup. See docs for more detailed setup instructions

# Start the policy server
./scripts/start_drtc_server.sh 

# Start the client 
./scripts/run_drtc_experiment_with_remote_server.sh --config baseline.yaml --remote-server-host (your policy server ip on same network)

• Extensive manual testing of the main DRTC code was conducted as described in the DRTC experiments although these experiments where run before recent tidy up changes to update my branch with the latest lerobot main branch

Unit Tests

• uv run pytest tests/async_inference/test_lww_register.py
• uv run pytest tests/async_inference/test_rtc_guidance.py
• uv run pytest tests/async_inference/test_rtc_transport.py

Checklist (required before merge)

• Linting/formatting run (pre-commit run -a)
• All tests pass locally (pytest)
• Documentation updated
• CI is green

Reviewer notes

• Anything the reviewer should focus on (performance, edge-cases, specific files) or general notes.
• Anyone in the community is free to review the PR.